To run the tests locally you can either run them in Xcode or through fastlane. The latter takes actual screenshots and require some additional setup: https://github.com/mullvad/mullvadvpn-app/tree/main/ios#screenshots-for-appstore

Running the tests in Xcode require only this configuration step of above: https://github.com/mullvad/mullvadvpn-app/tree/main/ios#configuration

Reviewable status: 0 of 5 files reviewed, all discussions resolved

Are the screenshot tests ran as part of the regular test suite? If not, we should change the regular ios.yml workflow to execute those tests too.

Reviewed 2 of 5 files at r1, all commit messages.
Reviewable status: 2 of 5 files reviewed, 5 unresolved discussions (waiting on @rablador)

.github/workflows/ios-screenshots.yml line 9 at r1 (raw file):

      - ios/.swiftformat
      - ios/**/*.swift
      - ios/**/*.xctestplan

When I tag releases, I often change just a single XCConfig and the Assets/changes.txt files, and since the all the push condiitons need to be satisfied, screenshots would never be ran.

.github/workflows/ios-screenshots.yml line 16 at r1 (raw file):

  check-formatting:
    name: Check formatting
    runs-on: macos-13-xlarge

I don't believe we need to run this job for screenshots.

.github/workflows/ios-screenshots.yml line 33 at r1 (raw file):

  swiftlint:
    name: Run swiftlint

Nor should we run swiftlint here too.

.github/workflows/ios-screenshots.yml line 48 at r1 (raw file):

  test:
    name: Take screenshots
    runs-on: macos-13-xlarge

We should try out different instance sizes and see if a smaller instance would work just as well - this job won't ever be blocking a merge, so it isn't critical that these tests must finish fast.

.github/workflows/ios-screenshots.yml line 99 at r1 (raw file):

          brew install xcbeautify

      - name: Run screenshot tests

A more apt name would be Create screenshots, I think.

ios/Snapfile line 6 at r1 (raw file):

  "iPhone 11 Pro Max", # 6.5"
  "iPhone 14 Pro Max", # 6.7"
  "iPad Pro (12.9-inch) (2nd generation)",

I'll double check this tomorrow, but we might want the older iPad too.

However, if running the screenshot test itself would take far too long, then we can run it on a slower instance once a day instead of doing it for every commit 🤷 .

Reviewable status: 2 of 5 files reviewed, 5 unresolved discussions (waiting on @rablador)

The iOS changes look fine to me

Reviewed 4 of 5 files at r1, all commit messages.
Reviewable status: 4 of 5 files reviewed, 5 unresolved discussions (waiting on @rablador)

Reviewable status: 4 of 5 files reviewed, 5 unresolved discussions (waiting on @rablador)

ios/Snapfile line 6 at r1 (raw file):

Yep, we need both the 6th gen and 2nd gen 12.9" iPad Pros.

After discussing with @pinkisemils IRL we decided to run the screenshot tests in the same github action. They add about a minute to the action overall.

Reviewable status: 3 of 6 files reviewed, 5 unresolved discussions (waiting on @buggmagnet and @pinkisemils)

.github/workflows/ios-screenshots.yml line 9 at r1 (raw file):

Done.

.github/workflows/ios-screenshots.yml line 16 at r1 (raw file):

Done.

.github/workflows/ios-screenshots.yml line 33 at r1 (raw file):

Done.

.github/workflows/ios-screenshots.yml line 99 at r1 (raw file):

Done

ios/Snapfile line 6 at r1 (raw file):

There are the ones we can get from github:

iPad Air (5th generation)
iPad (10th generation)
iPad mini (6th generation)
iPad Pro (11-inch) (4th generation)
iPad Pro (12.9-inch) (6th generation)

After IRL discussion we settled on adding iPad Pro (11-inch) (4th generation) as well.

Reviewed 1 of 5 files at r1, 3 of 3 files at r3, all commit messages.
Reviewable status: complete! all files reviewed, all discussions resolved

.github/workflows/ios-screenshots.yml line 48 at r1 (raw file):

I tried running the regular tests on the slower intel mac, and it's slower than it is cheap, so it'd cost about the same.

Let's merge this :)

Reviewable status: complete! all files reviewed, all discussions resolved

Reviewable status: 5 of 6 files reviewed, all discussions resolved (waiting on @pinkisemils)

.github/workflows/ios-screenshots.yml line 48 at r1 (raw file):

Running fastlane on the older runner doesn't work. Discussed IRL, so I'll set this discussion to resolved.

Reviewed 1 of 1 files at r4, all commit messages.
Reviewable status: complete! all files reviewed, all discussions resolved